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W. use data from oral reading fluency (ORF) assessments to examine COVID-19's effects on 
children’s ORF in over 100 U.S. school districts. Students’ development of ORF largely stopped in 
spring 2020 following the onset of the COVID-19 pandemic. In fall 2020, students’ gains in reading 
were stronger and similar to prepandemic rates. However, fall gains were insufficient to recoup 
spring losses; overall, students’ ORF in second and third grade Is approximately 50 percent behind 
expectations. We also observe inequitable impact: students at lower achieving schools are falling 
farther behind and 10 percent of students were not assessed this fall. While growth in ORF was 
stronger in the fall than in the spring, measures to address accumulated learning losses and to 
support students falling behind are needed. 
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Introduction 


Since the pandemic physically closed schools in March 2020, researchers have worked 
to predict! and then estimate the impact of extended educational disruptions on student 
learning. These studies have collectively raised the alarm on the issue of “learning loss"—the idea 
that students have learned less during the pandemic than in prior years. To this end, emerging 
research has shown that the pandemic has resulted in significant setbacks in English Language 
Arts and Math;? has exacerbated achievement gaps for low-income students and those learning 
English;? and may result in lost wages’ and lower life expectancy? in the long term. 


We build on this existing research by focusing on students in the earliest grades 
(Grades 1-3), who are often not included in existing learning loss analyses,® and by investigating 
the impact of the pandemic on student learning separately in spring 2020 and fall 2020. We 
address student learning in those critical early years by using data from an oral reading fluency 
(ORF) assessment—a rapid assessment taking only a few minutes that measures a fundamental 
reading skill-—to examine COVID-19's effects on children’s ORF during the pandemic in over 
100 U.S. school districts nationwide. 


The results in this brief and a related working paper’ can help shed light on the impact 
of the pandemic on student literacy development in the early grades.® Such evidence is unique 
given the relative paucity of testing in these early grades and can be used to inform decisions 
being made now about how resources should be allocated at the federal and state level to 
address student learning impacts related to the pandemic. 


Assessing COVID-19's Impact in the Early Grades Using Oral Reading Fluency 


Reading is perhaps the most important skill developed in K-12 education. ORF—the ability to 
read aloud quickly and accurately—is an important skill for developing readers. It can be measured 
relatively rapidly and is highly predictive of comprehension and other important measures of 
student reading ability.? Early literacy has been identified as a critical foundation for students’ future 
academic and life success."° 


To measure the effect of the pandemic on student development in this critical area, 
we use data from an assessment delivered by Literably (literably.com/learn) that measures ORF 
based on first recording students’ readings of texts presented to them on a device and then 
using a combination of human transcription and speech recognition to score these recordings. 
While these assessments are normally recorded in person in a classroom setting, students 
were able to take these recordings remotely during the COVID-19 pandemic, thus allowing us 
to study changes in growth of ORF related to disruptions in schooling using data from over 
250,000 scores collected in over 100 school districts soread across 22 states." 
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The Importance of Measuring Learning Changes in Both Spring and Fall 


Our analysis offers unique evidence about student growth in ORF throughout the various 
phases of the pandemic, looking both at spring 2020 and fall 2020 growth patterns to investigate 
when learning losses have been most acute. This is particularly critical given what we Know about 
educational experiences for children in these separate periods: in spring 2020 and through the 
summer, most U.S. students were sheltering at home with many receiving very little instruction; 
this fall, while the mode of instruction was quite varied both within and across states, most 
school districts were able to provide a higher quality of instruction, whether delivered in person 
or remotely.” 


Figure 1 provides a conceptual depiction of our analytic approach to measuring pandemic- 
related learning losses. Many existing studies Compare point-to-point (typically fall-to-fall) changes 
in student performance during COVID-19 to changes in performance over the same time period 
prior to COVID-19; that is, they compare changes in endpoints of yellow and blue dotted lines 
in Figure 1 (Panel A). In contrast, by utilizing Literably’s ongoing assessments, we can observe 
how the pandemic affected student learning throughout the year (Panel B). The measurable, 
accumulated learning loss at the end of fall 2020 is the same in Panels A and B, but the approach 
in Panel B allows for a more nuanced analysis. Specifically, the dotted blue line in Panel A suggests 
that the pandemic caused a uniform decrease in growth that has persisted across time, whereas 
the blue curve in Panel B reveals potential differences in learning growth across the year. Looking 
at the data this way provides more information about when learning losses occurred, and thus 
points towards more effective solutions. 


Figure 1. Conceptual Model: Using Ongoing Assessment to Understand Learning Changes 
Throughout the Pandemic 
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n the data used in our analysis, students in Grades 1-4 take three ORF assessments per 
year on average. We use change across these measures from before the pandemic to establish a 
baseline for “typical” growth in ORF. In a normal educational setting (i.¢., prior to the pandemic), 
students develop one additional word of ORF per minute (WPM) roughly every 10 days in Grades 
1 and 2; older students tend to develop ORF at a slower rate. Against this baseline level of growth, 
we Can compare growth measured during the COVID-affected period. 


Our analysis reveals three key findings: (a) growth in ORF stalled in the spring, with 
Grades 2 and 3 most affected; (b) growth in ORF picked up in the fall for tested students 
but was insufficient to make up for losses in the spring; and (c) students at lower achieving 
schools and those not taking the assessment may be at heightened risk of learning loss. 


Findings 


Growth in Oral Reading Fluency Stalled in the Spring, With Grades 2 and 3 Most Affected 


We illustrate the growth of ORF skills here in two ways. First, we examine the long-run 
growth of children who were in Grade 1 during 2018-19 (Figure 2). 


Figure 2. Estimated Within-Student Change in Oral Reading Fluency Over Several Years for a 
Single Cohort 
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For these students, who are in Grade 3 in 2020-21, growth was fairly steady until the onset 
of the pandemic (demarcated by the vertical red line, at which point they were in second grade). 
However, growth flattened in the spring of 2020 as the pandemic initially disrupted schooling’ 
and remained stagnant through the summer (the gray shaded column); during these periods, 
children’s reading abilities are not increasing. Relative to the extrapolated trajectory of growth 
(yellow line), these students have clearly suffered learning loss. Such an analysis is suggestive; 
we refine it below to better account for age-related changes in ORF growth. 


To provide another perspective that can adjust for other factors, such as changes in 
patterns of growth across the academic year, we also look at comparisons to growth for children 
in the same grade in unaffected years (Figure 3). In Grades 1-3, we show growth of students 
starting in the given grade in an unaffected year (2018-19; yellow line) and follow them into the 
following year. In the affected year (2019-20; blue line), same-grade students who experienced 
the pandemic showed similar patterns of growth up until the onset of the pandemic (demarcated 
by vertical red line). Following onset, growth slows during spring and summer 2020 before 
accelerating in the fall of 2020." 


Figure 3. Estimated Within-Student Change in Oral Reading Fluency Over 15 Months, Before and 
During the Pandemic 
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Growth in Oral Reading Fluency Picked Up in the Fall for Tested Students But Was 
Insufficient to Make Up for Losses from the Spring 


After the initial decline in ORF development in the spring of 2020, growth during the fall of 
2020 resumed at levels similar to before the pandemic. This effect can be seen above in Figure 3, 
where the learning rates (the slope of the blue line) are relatively similar before the pandemic and 
in fall 2020, but with gaps in learning due to the stagnation of learning in spring 2020. To explore 
this effect further, we consider growth rates during the fall and compare them across academic 
years (Figure 4). Yellow bars show growth rates for the first 90 days of 2019-20, prior to the 
onset of the pandemic. Note that these growth rates are fairly hign compared to the year-long 
growth rates noted earlier (e.g., words per minute / day = 0.1); students typically develop ORF 
faster at the beginning of the year than at the end. Blue bars show growth rates for the first 90 
days of the COVID-affected 2020-21 year. In fall 2020, Grade 1 students seemed to learn at 
rates comparable to those observed for Grade 1 students prepandemic. In Grades 2-4, however, 
growth rates were somewhat lower in the fall during the pandemic (although not statistically 
different). For example, growth was lower in fall 2020 in Grade 2 than it was in past years; we 
observed a growth of 0.18 WPM per day in fall 2019 and growth of 0.16 WPM per day in fall 2020. 


While the fall learning rates are improvements over the stagnant growth from the previous 
spring, they are insufficient to make up for the lost ground students experienced earlier in the 
pandemic. The accumulated learning loss for second and third graders would leave them 7.3 and 
7.7 WPM behind their expected level respectively (representing 26 percent and 33 percent of the 
expected yearly gains). Further, we have reason to believe these are underestimates of the true 
learning loss, as many students were not assessed in spring 2020." If unassessed students were 
not provided instructional opportunities or were otherwise not engaged in learning, their ORF 
could have been yet more severely impacted. 


Figure 4. Grade-Specific Fall Growth Rates Across Academic Years 
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New Concerns Are Emerging Around Inequitable Impacts in Fall 2020: Students at Lower 
Achieving Schools Are Falling Farther Behind and Students Not Taking the Assessment May 
Be at Heightened Risk of Learning Loss 


In analyses of this kind, it is always critical to understand the differential impact on 
students. We emphasize two points of concern for how the pandemic may have inequitably 
affected students continuing into the fall.?° First, students in lower achieving districts are perhaps 
falling farther behind their peers in higher achieving districts. When we compare fall 2020 
growth” in districts with higher annual state test scores to growth in lower achieving districts 
(using the Stanford Education Data Archive from the Educational Opportunity Project site: 
edopportunity.org), we observe more robust growth in ORF in higher achieving districts. Figure 
5 shows projected growth over time for students in districts at the 10th and 90th percentile of 
the achievement scale. The slopes of the lines are steeper for higher achieving districts; students 
in a 90th-percentile district would be expected to gain 20.4, 19.1, and 13.4 WPM over the fall; in 
contrast, students in a 1Oth-percentile district would be expected to gain 12.5, 10.5, and 8.6 WPM 
respectively. These gaps in growth between high- and low-achieving districts are not generally 
observed In previous years, suggesting that they are emerging as a consequence of the COVID-19 
pandemic and, if left unchecked, will result in a widening of preexisting achievement gaps. 


Figure 5. Comparison of Within-Person Change in Oral Reading Fluency for Districts at 10th and 
90th Percentile of Achievement on Annual Standardized Tests 
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Second, students who are not able to take the assessment in fall 2020 may be falling 
farther behind. We estimate that approximately 10 percent of the students who were present in 
schools in 2019-20 were unobserved in fall 2020, but we do not have information about why 
these students were missing. If they have trouble accessing remote assessment—for example 
because they do not have reliable access to computers or Wi-Fi®—they may also be engaging 
less with distance learning. There is emerging evidence that low-income students have been 
absent more often’? or have simply not reenrolled in school?° during the pandemic, raising equity 
concerns both about who is assessed and what kinds of instructional support they are receiving 
during the pandemic. Although our analysis reveals that prior ORF scores do not predict whether 
a student is assessed in fall 2020*-meaning that the students with the lowest levels of prior 
ORF are roughly as likely as others to be missing—we suspect that students missing from our 
analysis may have experienced even larger learning losses than those students we did observe. 


Conclusion 


In this analysis we examine the effect of the COVID-19 pandemic on children’s ORF. 
We find that, overall, students’ ORF in second and third grade Is approximately 30 percent behind 
what we would expect in a typical year. Furthermore, given that many students were not assessed, 
these averages may be underestimates of true losses. Our analyses also point to continued 
reasons for concern: students in lower achieving districts experienced greater declines in learning 
in the fall that will widen existing achievement gaps if left unaddressed. 


Implications for Policy and Practice 


This interruption in growth of ORF as a result of the pandemic in primary grades is 
concerning given the fundamental nature of reading for academic development. A large body 
of prior research has identified literacy by the third grade as critical for students’ future academic 
and life success.” If students are unable to regain this loss, they may experience delays in other 
reading-related skill building, making it difficult to access future academic content and learning. 


We are particularly concerned about the inequitable impact of the pandemic on 
student learning. Our analysis reveals that students in lower-achieving districts are falling behind 
faster relative to others. This demands attention from policymakers and educators, as greater 
learning loss for students in lower achieving school districts may exacerbate disparities. As 
leaders at the national and local level ponder policies that address the impact of the pandemic 
on students and families, we recommend that substantial resources be allocated to support 
literacy development in the early grades. Even greater resources need to be shared with low- 
achieving districts, which often serve a disproportionate number of low-income students. 
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Our research demonstrates that, after the initial shock of the pandemic in the spring, 
learning has resumed this fall. This provides evidence that educators are finding ways to 
successfully teach and assess ORF even during the continued disruptions and uncertainty of 
the pandemic. We can build on this success to make sure that educators have the appropriate 
resources, especially in schools serving students with the highest need. In addition to targeted 
funding, it will be important to identify the practices that accelerate learning for students 
who have fallen behind, and to build policy and support structures that help to enact these 
practices at scale. 


Limitations and Next Steps 


This analysis, while providing crucial evidence about the impact of the pandemic on the 
development of basic skills in the early grades, leaves critical questions unanswered. First, Our 
analysis tracks students across 100 school districts in 22 states but does not currently include 
information about how students were educated. There is tremendous variation across and within 
states as to whether schools served students in fall 2020 in person, remotely, or in a hybrid 
fashion,*3 and plans have continued to shift based on local transmission rates, changing safety 
protocols, and the political environment. Future analysis will investigate differences in outcomes 
in fall 2020 based on mode of instructional delivery. 


Second, additional research should be done to further understand the pandemic’s impact 
on students who have been missing from school during the pandemic. Based on our analysis, 
70 percent of students who tested in fall 2019 did not test again in spring 2020 and we suspect 
that 10 percent of students we would typically have observed did not test in fall 2020. Because 
these students are likely to be those least connected to educational opportunity, it is probable 
that our estimates of learning losses in both fall and spring 2020 are an underestimate of the 
true average effect. Even more importantly, if losses for these specific students are even more 
dramatic than what has been observed, they may have more difficulty catching up and will suffer 
the consequences well into the future. Educators and researchers alike must make concerted 
efforts first to understand the circumstances surrounding these students and then to reengage 
them and employ compensatory instructional practices to help them regain learning growth. 


Third, results might not generalize to other subjects. Reading is a focal point of instruction 
in early education and instructional adjustments related to reading were likely a centerpiece of 
many district's plans for instruction during the fall of 2020. Given its centrality, learning losses In 
reading may be qualitatively different from those in other subject areas. 


Finally, the full extent of learning loss during the pandemic will not be understood for 
months or even years. Gaps in ORF that emerge now may lead to gaps in other subjects over 
time if problems in students’ ORF interfere with content learning in later grades. And new gaps 
may emerge: for example, with enrollments down in preschool and kindergarten programs 
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this year,*4 it is possible that incoming students in 2021-22 will start behind. As such, we must 
continue to examine and monitor academic performance, learning loss, and renewed learning 
growth as students continue in distance learning and return to school. In our ongoing research, 
we will continue to monitor ORF as educational policies and practices continue to adjust as we 
recover from the pandemic. 
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